Neural Relighting. Instead of modeling BRDF parameters, learning-based relighting approaches attempt to directly learn relightable appearance from a light-stage capture [85, 48, 13, 93, 49, 67, 84]. While these approaches show promising relighting for static [85, 93, 67, 84] and dynamic scenes [48, 49], they do not support generating novel animations, which is an essential requirement for avatars. Portrait relighting methods [71, 72, 77, 58, 89] also enable relighting under novel illuminations given a single image. However, they cannot produce novel view synthesis or temporally coherent dynamic relighting. Bi et al. [6] propose a neural-rendering method that supports global illumination for animatable facial avatars. To enable real-time rendering with natural environments, they distillate a slow teacher model conditioned with individual point lights into a light-weight student model that can be conditioned with environment maps. This work is later extended to articulated hand modeling [20], compositional modeling of heads and eyeglasses [35], and scalable training by eliminating the need of teacher-student distillation [88]. These relightable avatars take as input the lighting information, which we discover is the main limiting factor for expressiveness in all-frequency relighting. In contrast, inspired by Precomputed Radiance Transfer (PRT) [70, 75], we propose to integrate a target illumination at the output of our neural decoder, improving quality and simplifying the learning pipeline. Precomputed Radiance Transfer. In computer graphics, rendering a scene with global illumination is an expensive process due to iterative path tracing or multiple bounce computations. To enable real-time rendering with global light transport, Sloan et al. [70] propose to precompute a part of light transport that only depends on intrinsic scene properties, such as geometry and reflectance, and then integrate the precomputed intrinsic factor with an extrinsic illumination at runtime. For fast integration, they utilize spherical harmonics as an angular basis. To overcome the limited frequency band in spherical harmonics, follow-up works introduce wavelets [56], spherical radial basis functions [74], spherical Gaussians [17, 75], anisotropic spherical Gaussians [82], and neural network-based decompositions [86]. Similarly, Neural PRT [63] applies the same principle to screen-space relighting based on neural deferred rendering [73]. Despite many desirable properties, these methods primarily focus on static scenes due to the dependency on knowing geometry and material properties. Unfortunately, we neither know the geometry and material properties for human heads a priori, nor are they static. Thus, we propose to learn the intrinsic radiance transfer from dynamic real-data observations without explicitly assuming any material types or underlying geometry. The closest to our work in terms of the appearance representation is EyeNeRF [33], where they learn view-independent spherical harmonics for diffuse and view-conditioned spherical harmonics for specular components from image observations to build a relightable eye model. However, this appearance model suffers from the limited expressiveness of spherical harmonics for specular reflections. Empirically, we find that their proposed model does not generalize well to novel view and light directions. Please refer to Sec. 4 for our analysis.